{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Getting started with Lasagne\n",
    "\n",
    "This workbook contains the code for this blog post:\n",
    "http://luizgh.github.io/libraries/2015/12/08/getting-started-with-lasagne\n",
    "\n",
    "Here we train a CNN on the MNIST dataset using the Lasagne library (that needs to be installed before running this workbook)\n",
    "http://lasagne.readthedocs.org/en/latest/user/installation.html\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using gpu device 0: Tesla C2050 (CNMeM is disabled)\n"
     ]
    }
   ],
   "source": [
    "#For running the model:\n",
    "import lasagne\n",
    "import theano\n",
    "from theano import tensor as T\n",
    "import numpy as np\n",
    "\n",
    "#For loading the dataset:\n",
    "import os\n",
    "import gzip\n",
    "import cPickle\n",
    "import urllib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define input / output shape and symbolic variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data_size=(None,1,28,28) # Batch size x Img Channels x Height x Width\n",
    "output_size=10 # We will run the example in mnist - 10 digits\n",
    "\n",
    "input_var = T.tensor4('input')\n",
    "target_var = T.ivector('targets')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Define the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "net = {}\n",
    "\n",
    "#Input layer:\n",
    "net['data'] = lasagne.layers.InputLayer(data_size, input_var=input_var)\n",
    "\n",
    "#Convolution + Pooling\n",
    "net['conv1'] = lasagne.layers.Conv2DLayer(net['data'], num_filters=6, filter_size=5)\n",
    "net['pool1'] = lasagne.layers.Pool2DLayer(net['conv1'], pool_size=2)\n",
    "\n",
    "net['conv2'] = lasagne.layers.Conv2DLayer(net['pool1'], num_filters=10, filter_size=5)\n",
    "net['pool2'] = lasagne.layers.Pool2DLayer(net['conv2'], pool_size=2)\n",
    "\n",
    "\n",
    "#Fully-connected + dropout\n",
    "net['fc1'] = lasagne.layers.DenseLayer(net['pool2'], num_units=100)\n",
    "net['drop1'] = lasagne.layers.DropoutLayer(net['fc1'],  p=0.5)\n",
    "\n",
    "#Output layer:\n",
    "net['out'] = lasagne.layers.DenseLayer(net['drop1'], num_units=output_size, \n",
    "                                       nonlinearity=lasagne.nonlinearities.softmax)\n",
    "\n",
    "#Note: Lasagne does not implement a class for \"Model\". Usually, you only need the output layer to:\n",
    "# 1) Obtain the result of the layer (predictions)\n",
    "# 2) Obtain a list of all parameters from the model (e.g. for weight decay)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Define the cost function and update rule"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Define hyperparameters. These could also be symbolic variables \n",
    "lr = 1e-2\n",
    "weight_decay = 1e-5\n",
    "\n",
    "\n",
    "#Loss function: mean cross-entropy\n",
    "prediction = lasagne.layers.get_output(net['out'])\n",
    "loss = lasagne.objectives.categorical_crossentropy(prediction, target_var)\n",
    "loss = loss.mean()\n",
    "\n",
    "#Also add weight decay to the cost function\n",
    "weightsl2 = lasagne.regularization.regularize_network_params(net['out'], lasagne.regularization.l2)\n",
    "loss += weight_decay * weightsl2\n",
    "\n",
    "#Get the update rule. Here we will use a more advanced optimization algorithm: ADAM [1]\n",
    "params = lasagne.layers.get_all_params(net['out'], trainable=True)\n",
    "updates = lasagne.updates.adam(loss, params)\n",
    "\n",
    "#[1] http://arxiv.org/abs/1412.6980"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Compile the training and test functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_prediction = lasagne.layers.get_output(net['out'], deterministic=True)\n",
    "test_loss = lasagne.objectives.categorical_crossentropy(test_prediction,\n",
    "                                                        target_var)\n",
    "test_loss = test_loss.mean()\n",
    "test_acc = T.mean(T.eq(T.argmax(test_prediction, axis=1), target_var),\n",
    "                  dtype=theano.config.floatX)\n",
    "\n",
    "#Note that train_fn has a \"updates\" rule. Whenever we call this function, it updates the parameters of the model.\n",
    "train_fn = theano.function([input_var, target_var], loss, updates=updates, name='train')\n",
    "val_fn = theano.function([input_var, target_var], [test_loss, test_acc], name='validation')\n",
    "get_preds = theano.function([input_var], test_prediction, name='get_preds')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtain dataset (MNIST)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dataset_file = 'mnist.pkl.gz'\n",
    "\n",
    "#Download dataset if not yet done:\n",
    "if not os.path.isfile(dataset_file):\n",
    "    urllib.urlretrieve('http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist.pkl.gz', dataset_file)\n",
    "\n",
    "#Load the dataset\n",
    "f = gzip.open(dataset_file, 'rb')\n",
    "train_set, valid_set, test_set = cPickle.load(f)\n",
    "f.close()\n",
    "\n",
    "#Convert the dataset to the shape we want\n",
    "x_train, y_train = train_set\n",
    "x_test, y_test = test_set\n",
    "\n",
    "x_train = x_train.reshape(-1, 1, 28, 28)\n",
    "x_test = x_test.reshape(-1, 1, 28, 28)\n",
    "y_train = y_train.astype(np.int32)\n",
    "y_test = y_test.astype(np.int32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50, train error: 0.551961. Elapsed time: 9.81 seconds\n",
      "Epoch 2/50, train error: 0.205821. Elapsed time: 9.81 seconds\n",
      "Epoch 3/50, train error: 0.152390. Elapsed time: 9.81 seconds\n",
      "Epoch 4/50, train error: 0.125015. Elapsed time: 9.82 seconds\n",
      "Epoch 5/50, train error: 0.108658. Elapsed time: 9.81 seconds\n",
      "Epoch 6/50, train error: 0.097270. Elapsed time: 9.82 seconds\n",
      "Epoch 7/50, train error: 0.090378. Elapsed time: 9.81 seconds\n",
      "Epoch 8/50, train error: 0.084239. Elapsed time: 9.81 seconds\n",
      "Epoch 9/50, train error: 0.077371. Elapsed time: 9.81 seconds\n",
      "Epoch 10/50, train error: 0.073579. Elapsed time: 9.82 seconds\n",
      "Epoch 11/50, train error: 0.069367. Elapsed time: 9.81 seconds\n",
      "Epoch 12/50, train error: 0.065499. Elapsed time: 9.83 seconds\n",
      "Epoch 13/50, train error: 0.063542. Elapsed time: 9.82 seconds\n",
      "Epoch 14/50, train error: 0.058625. Elapsed time: 9.81 seconds\n",
      "Epoch 15/50, train error: 0.056276. Elapsed time: 9.81 seconds\n",
      "Epoch 16/50, train error: 0.056554. Elapsed time: 9.80 seconds\n",
      "Epoch 17/50, train error: 0.053809. Elapsed time: 9.81 seconds\n",
      "Epoch 18/50, train error: 0.052501. Elapsed time: 9.81 seconds\n",
      "Epoch 19/50, train error: 0.049518. Elapsed time: 9.81 seconds\n",
      "Epoch 20/50, train error: 0.048927. Elapsed time: 9.81 seconds\n",
      "Epoch 21/50, train error: 0.048280. Elapsed time: 9.81 seconds\n",
      "Epoch 22/50, train error: 0.046336. Elapsed time: 9.81 seconds\n",
      "Epoch 23/50, train error: 0.046282. Elapsed time: 9.80 seconds\n",
      "Epoch 24/50, train error: 0.044631. Elapsed time: 9.81 seconds\n",
      "Epoch 25/50, train error: 0.043654. Elapsed time: 9.82 seconds\n",
      "Epoch 26/50, train error: 0.043019. Elapsed time: 9.81 seconds\n",
      "Epoch 27/50, train error: 0.043541. Elapsed time: 9.82 seconds\n",
      "Epoch 28/50, train error: 0.043016. Elapsed time: 9.82 seconds\n",
      "Epoch 29/50, train error: 0.038510. Elapsed time: 9.82 seconds\n",
      "Epoch 30/50, train error: 0.038796. Elapsed time: 9.80 seconds\n",
      "Epoch 31/50, train error: 0.039463. Elapsed time: 9.81 seconds\n",
      "Epoch 32/50, train error: 0.038776. Elapsed time: 9.81 seconds\n",
      "Epoch 33/50, train error: 0.038409. Elapsed time: 9.81 seconds\n",
      "Epoch 34/50, train error: 0.037471. Elapsed time: 9.81 seconds\n",
      "Epoch 35/50, train error: 0.034132. Elapsed time: 9.80 seconds\n",
      "Epoch 36/50, train error: 0.035962. Elapsed time: 9.82 seconds\n",
      "Epoch 37/50, train error: 0.036778. Elapsed time: 9.82 seconds\n",
      "Epoch 38/50, train error: 0.034642. Elapsed time: 9.80 seconds\n",
      "Epoch 39/50, train error: 0.034611. Elapsed time: 9.82 seconds\n",
      "Epoch 40/50, train error: 0.033472. Elapsed time: 9.81 seconds\n",
      "Epoch 41/50, train error: 0.032992. Elapsed time: 9.81 seconds\n",
      "Epoch 42/50, train error: 0.034621. Elapsed time: 9.81 seconds\n",
      "Epoch 43/50, train error: 0.032304. Elapsed time: 9.82 seconds\n",
      "Epoch 44/50, train error: 0.032512. Elapsed time: 9.81 seconds\n",
      "Epoch 45/50, train error: 0.031692. Elapsed time: 9.80 seconds\n",
      "Epoch 46/50, train error: 0.031167. Elapsed time: 9.81 seconds\n",
      "Epoch 47/50, train error: 0.032137. Elapsed time: 9.80 seconds\n",
      "Epoch 48/50, train error: 0.031676. Elapsed time: 9.79 seconds\n",
      "Epoch 49/50, train error: 0.031260. Elapsed time: 9.81 seconds\n",
      "Epoch 50/50, train error: 0.032171. Elapsed time: 9.81 seconds\n",
      "Training completed in 490.58 seconds.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "epochs = 50  #You can reduce the number of epochs to run it  faster (or run it for longer for a better model)\n",
    "batch_size=100\n",
    "\n",
    "#Run the training function per mini-batches\n",
    "n_examples = x_train.shape[0]\n",
    "n_batches = n_examples / batch_size\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "cost_history = []\n",
    "for epoch in xrange(epochs):\n",
    "    st = time.time()\n",
    "    batch_cost_history = []\n",
    "    for batch in xrange(n_batches):\n",
    "        x_batch = x_train[batch*batch_size: (batch+1) * batch_size]\n",
    "        y_batch = y_train[batch*batch_size: (batch+1) * batch_size]\n",
    "        \n",
    "        this_cost = train_fn(x_batch, y_batch) # This is where the model gets updated\n",
    "        \n",
    "        batch_cost_history.append(this_cost)\n",
    "    epoch_cost = np.mean(batch_cost_history)\n",
    "    cost_history.append(epoch_cost)\n",
    "    en = time.time()\n",
    "    print('Epoch %d/%d, train error: %f. Elapsed time: %.2f seconds' % (epoch+1, epochs, epoch_cost, en-st))\n",
    "    \n",
    "end_time = time.time()\n",
    "print('Training completed in %.2f seconds.' % (end_time - start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test error: 0.008200\n",
      "Classifying 10000 images completed in 0.52 seconds.\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "loss, acc = val_fn(x_test, y_test)\n",
    "test_error = 1 - acc\n",
    "print('Test error: %f' % test_error)\n",
    "\n",
    "end_time = time.time()\n",
    "print('Classifying %d images completed in %.2f seconds.' % (x_test.shape[0], end_time - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fc0ceee46d0>]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEACAYAAABMEua6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAF1xJREFUeJzt3X2QXXWd5/H3Nx1iEghghAGBIA9GngYQLAOjiO3wsIFa\nhdWVxx0p2EFqa9mxtmYohq0dTdXOjMXglJTFrMvuIGU5lMFZdyDOypOuvYqDPAmIEDBZyEACJCAE\n8pzu5Lt//G7TN02n7+2m70Of+35Vnbrn3Hv6nh+/oj7nl+/5nXMjM5EkVcOMTjdAkjR1DHVJqhBD\nXZIqxFCXpAox1CWpQgx1SaqQhqEeEYsj4pmIWBER1+5mn/6IeCwifh0RA1PeSklSU2K8eeoR0Qc8\nC5wJrAEeBi7OzOV1++wL/Bz4F5m5OiL2y8zXWttsSdJYGo3UFwErM3NVZg4CS4HzRu1zCfD9zFwN\nYKBLUuc0CvWDgRfrtlfX3qu3EJgfET+JiEci4g+msoGSpObNbPB5M88Q2AM4GTgDmAs8EBG/yMwV\n77ZxkqSJaRTqa4AFddsLKKP1ei8Cr2XmFmBLRPwUOBHYJdQjwofMSNIkZGY0u2+j8ssjwMKIOCwi\nZgEXAstG7XMncFpE9EXEXOAU4OndNMwlk6985Ssdb0O3LPaFfWFfjL9M1Lgj9cwcioirgXuAPuCW\nzFweEVfVPr85M5+JiLuBXwE7gf+RmWOGuiSptRqVX8jMu4C7Rr1386jtrwFfm9qmSZImyjtKO6C/\nv7/TTega9sUI+2KEfTF54958NKUHish2HUuSqiIiyCm8UCpJmkYMdUmqEENdkirEUJekCjHUJalC\nDHVJqhBDXZIqxFCXpAox1CWpQgx1SaoQQ12SKsRQl6QKMdQlqUIMdUmqEENdkirEUJekCjHUJalC\nDHVJqhBDXZIqxFCXpAox1CWpQgx1SaqQtob69u3tPJok9Z62hvqGDe08miT1HkNdkiqkraH+1lvt\nPJok9Z6GoR4RiyPimYhYERHXjvF5f0S8GRGP1Zb/vLvvcqQuSa01c7wPI6IPuAk4E1gDPBwRyzJz\n+ahd/29mfqbRwQx1SWqtRiP1RcDKzFyVmYPAUuC8MfaLZg5m+UWSWqtRqB8MvFi3vbr2Xr0EPhYR\nT0TEDyPi2N19mSN1SWqtccsvlMBu5JfAgszcHBHnAHcAHxprR0NdklqrUaivARbUbS+gjNbflpkb\n6tbvioj/GhHzM/P10V92551LWL++rPf399Pf3z/JZktSNQ0MDDAwMDDpv4/M3Q/GI2Im8CxwBvAS\n8BBwcf2F0og4AFiXmRkRi4DvZeZhY3xX/smfJDfcMOm2SlLPiQgys6nrltBgpJ6ZQxFxNXAP0Afc\nkpnLI+Kq2uc3A/8a+HcRMQRsBi7a3fdZfpGk1hp3pD6lB4rISy5JbrutLYeTpEqY6EjdO0olqUJ8\n9oskVYihLkkVYvlFkirEkbokVYihLkkV0tZQ37oVhobaeURJ6i1tDfW99oKNG9t5REnqLW0N9b33\ntgQjSa3U1lCfN88ZMJLUSm0PdUfqktQ6ll8kqUIsv0hShVh+kaQKsfwiSRXiSF2SKsSauiRViOUX\nSaoQyy+SVCGWXySpQhypS1KFWFOXpAqx/CJJFWL5RZIqxPKLJFVIZGZ7DhSRO3Yke+wBg4Mwo62n\nE0maniKCzIxm929rtM6YAXPmwKZN7TyqJPWOto+XLcFIUuu0PdSdASNJrdMw1CNicUQ8ExErIuLa\ncfb7aEQMRcRnx/s+Z8BIUuuMG+oR0QfcBCwGjgUujohjdrPf9cDdwLgFfcsvktQ6jUbqi4CVmbkq\nMweBpcB5Y+z3H4D/Cbza6ICO1CWpdRqF+sHAi3Xbq2vvvS0iDqYE/Tdrb407R9KauiS1zswGnzcz\nif1G4E8zMyMiGKf8smTJEp59Ft54AxYs6Ke/v38CTZWk6hsYGGBgYGDSfz/uzUcRcSqwJDMX17av\nA3Zm5vV1+zzHSJDvB2wGrszMZaO+KzOTa6+F+fPh2t1ecpUkDZvozUeNRuqPAAsj4jDgJeBC4OL6\nHTLziLqD3wr8YHSg17P8IkmtM26oZ+ZQRFwN3AP0Abdk5vKIuKr2+c0TPeDee8O6dZNqqySpgUYj\ndTLzLuCuUe+NGeaZeXmj73P2iyS1jneUSlKFdCTUHalLUmv4QC9JqhDLL5JUIZZfJKlCLL9IUoW0\n9efsMpOhIZg9u/ykXTR9j5Qk9aau/jk7gJkzYY89YMuWdh9ZkqqvIz//bAlGklqjI6HuxVJJao2O\nhbrTGiVp6ll+kaQKsfwiSRVi+UWSKsTyiyRViOUXSaoQyy+SVCGWXySpQiy/SFKFWH6RpApxpC5J\nFWJNXZIqxPKLJFWI5RdJqhDLL5JUIY7UJalCOhLq73kPZMK2bZ04uiRVV0dCHSzBSFIrNAz1iFgc\nEc9ExIqIuHaMz8+LiCci4rGIeDQifr+ZA1uCkaSpN3O8DyOiD7gJOBNYAzwcEcsyc3ndbj/KzDtr\n+x8P/APwwUYHdlqjJE29RiP1RcDKzFyVmYPAUuC8+h0yc1Pd5l7Aa80c2PKLJE29RqF+MPBi3fbq\n2nu7iIjzI2I5cBfwR80c2PKLJE29ccsvQDbzJZl5B3BHRHwC+A5w1Fj7LVmy5O31LVv6eeut/qYa\nKUm9YmBggIGBgUn/fWTuPrcj4lRgSWYurm1fB+zMzOvH+Zv/ByzKzN+Oej/rj/WHfwinnlpeJUlj\niwgyM5rdv1H55RFgYUQcFhGzgAuBZaMOeGRERG39ZIDRgT4Wyy+SNPXGLb9k5lBEXA3cA/QBt2Tm\n8oi4qvb5zcDngC9ExCCwEbiomQM7+0WSpt645ZcpPdCo8svXvgavvFJeJUljm+ryS8tYfpGkqdfR\nULf8IklTy5G6JFWID/SSpApxpC5JFWJNXZIqxPKLJFWI5RdJqpCOhfqcObB9OwwNdaoFklQ9HQv1\nCEfrkjTVOhbqYKhL0lTreKg7A0aSpk5HQ90ZMJI0tTo+UjfUJWnqdDzULb9I0tSx/CJJFdLxkbqh\nLklTp+OhbvlFkqaO5RdJqpCOj9QNdUmaOoa6JFVIx0PdmrokTR1r6pJUIR0fqRvqkjR1Oh7qll8k\naepYfpGkCun4SN1Ql6SpE5nZngNF5Ohj7dwJe+wBg4Mwo6OnF0nqThFBZkaz+zcVpRGxOCKeiYgV\nEXHtGJ9fGhFPRMSvIuLnEXFCUwefAXvuCRs3NttcSdJ4GoZ6RPQBNwGLgWOBiyPimFG7PQecnpkn\nAP8F+O/NNsASjCRNnWZG6ouAlZm5KjMHgaXAefU7ZOYDmflmbfNB4JBmG3DoobB8ebN7S5LG00yo\nHwy8WLe9uvbe7vxb4IfNNuCzn4W///tm95YkjWdmE/s0fSU1Ij4FXAF8fKzPlyxZ8vZ6f38//f39\nfP7z8NGPwt/8DcxspjWSVGEDAwMMDAxM+u8bzn6JiFOBJZm5uLZ9HbAzM68ftd8JwP8CFmfmyjG+\n5x2zX4adcgr8+Z/DWWdN7j9CkqqqFbNfHgEWRsRhETELuBBYNuqgh1IC/d+MFeiNXHAB3H77RP9K\nkjRaU/PUI+Ic4EagD7glM78aEVcBZObNEfG3wL8CXqj9yWBmLhr1Hbsdqb/wApx8Mrz8cpm3Lkkq\nJjpS7+jNR/U+9jH48pdh8eK2NEeSpoWW3HzUDpZgJOnd65qR+urVcMIJ8MorMGtWW5okSV1v2o7U\nDzkEjjsO7ruv0y2RpOmra0IdSgnme9/rdCskafrqmvILwEsvldH6yy/D7NltaZYkdbVpW34BOOig\nUle/995Ot0SSpqeuCnWACy+0BCNJk9VV5Rcos1+OPrqUYObMaUPDJKmLTevyC8CBB5a7S+++u9Mt\nkaTpp+tCHZwFI0mT1XXlF4BXX4WFC8tsmLlzW9wwSepi0778ArD//uUZ6z9s+qc2JEnQpaEOPgtG\nkiajK8svAG+8AcccA8uWwaJFjfeXpCqqRPkF4L3vhRtvhMsvh23bOt0aSZoeujbUodyItHBh+ak7\nSVJjXVt+Gfbyy3DiiXDPPXDSSS1omCR1scqUX4a9//1www1wxRUwONjp1khSd+v6UAf4whfKnabX\nX9/plkhSd+v68suwF18sjw8YGCiP55WkXlC58suwBQvgL/6izIYZGup0aySpO02bUAe48kqYNw++\n/vVOt0SSutO0Kb8Me+65cjPSz38ORx01BQ2TpC5W2fLLsCOOgL/8SzjnHFixotOtkaTuMrPTDZiM\nL34RIuD00+Ef/xE+8pFOt0iSusO0G6kPu/JK+OY3YfFi+NGPOt0aSeoO0zbUAc4/H77/fbjkEn9U\nQ5JgmpZf6p1+ehmpn3surFsHV1/d6RZJUuc0NVKPiMUR8UxErIiIa8f4/OiIeCAitkbEH099M8d3\nwgnws5/BN74Bf/Zn0KYJPZLUdRpOaYyIPuBZ4ExgDfAwcHFmLq/bZ3/gA8D5wBuZ+ddjfM+UTGkc\nz7p18OlPwz77lHr7kUe29HCS1HKtmNK4CFiZmasycxBYCpxXv0NmvpqZjwAdfeTW7/wO3H8/nHUW\nnHIK/NVf+RAwSb2lmVA/GHixbnt17b2utMcecM018PDD8OMfl986ffjhTrdKktqjmVCflhXqww+H\nu+8uAf/pT8OXvgQbNnS6VZLUWs3MflkDLKjbXkAZrU/YkiVL3l7v7++nv79/Ml/TtAi49NIyl/2a\na+Doo8tz2S+/vNyZKkndZmBggIGBgUn/fTMXSmdSLpSeAbwEPMSoC6V1+y4BNnTqQmkjTz4J3/oW\n3HZbeXzvFVfA5z4Hc+d2tFmStFsTvVDa1AO9IuIc4EagD7glM78aEVcBZObNEXEgZVbM3sBOYANw\nbGZurPuOjof6sO3b4Qc/KAH/wAPw+c/DZZfB7/1eGd1LUrdoSahPhW4K9Xpr1sC3vw1/93ewaRNc\ncAFcdFH5QQ4DXlKnGeqTlAm//jXcfjssXVreu/DCshx/vAEvqTMM9SmQCb/8ZQn422+Hffct9fdL\nL4X99ut06yT1EkN9iu3cWX4X9dZbSx3+zDNLwJ99Nsyc9k/OkdTtDPUWevPNUpq59dbyQ9iXXgr9\n/eUGp/3373TrJFWRod4mTz0F3/0u/OIX5Y7V972v/Mze8HLyyU6VlPTuGeodsHMn/OY38NBDI8tT\nT8Hv/i6cdlpZPv7x8mwaSZoIQ71LbNlSRvD331+Wf/qnEuqnnVbmw596Khx7LPT1dbqlkrqZod6l\nduwoo/f77y8lmwcfhJdfLr+vesopZVm0CA46yOmTkkYY6tPI66+X0fyDD5blkUfKxdiDDy7LIYeM\nLEceCZ/4BOy9d6dbLamdDPVpbsuWcpfr6tW7Lk8/XWr1J59cplOefXYZ5Vu+karNUK+wzZvhpz+F\ne++F++6Dl16CM84opZvDDy/LEUeUX36SVA2Geg9Zs6aE++OPw/PPl+W552DWrBLwhx1W1nfsgKGh\nXZe+vlLSOeqokWXBApjR1K/WSmoXQ73HZcJrr5WAX7WqBHpfX7n7tX7Zvh1WroRnnx1Z1q+HD36w\nnAzmzy9z79/3vl3XjzgCDj3Ui7lSuxjqmrQNG8p8+xdeKBdxf/vbXV9fe62cCN56C445pkzJPO64\n8vqhD8G8eTBnDsyeXf6FYPBL756hrpZbv75cuB1ennoKVqwojy7esgW2bi0lntmzyzJnzsgyd+6u\n6/vtt+tMn+H1vff2pCCBoa4usWNHCfetW0vQb9lSLvQOrw9vv/pqmd0zPONn+HXnzvIj4sPBHjGy\nvOc95UauAw+EAw7Y9XX+/HJCqF/22stZQpq+DHVVwsaNMDhY1jNHFigninXr4JVXYO3aXV9ff72U\nkd56a2TZuPGd/0qoX2bNKtcYtm6Fbdt2fYVyEpk9u7wOr8+ZU56z399fHgExb15Hukk9wFCXRtm5\ns5SGRv9LYXjZvn3XwK5/hRLwo8N+0yZ49NHyWOZHHx0J+P7+MsV05sx3zjgaGirHGm5HfXs2by5/\nM3zimTt315PQrFnlXy7Dr8OLs5Wqz1CX2mzLlvLoh4GBkZCHd844mjmzBHF9cNe/7thRwr0+9IeX\nwcGybN8+sj44WL5v3rxSZpo3b9f1ww6DE08sy1FHlX3HMzRULopv3FiWTZt2fZ0zp8x8OvTQUu7y\nmkd7GOpSj8gsIT9cbtqwYdfS03PPlXsYnniiPP//mGPgwx8us5U2bSo3r9Uvr75afuVr3jzYc89y\nLaL+dcsW+Od/LrOjNmwo9zV84APldc6c0qb6oI8o1zL22ad8b/3y3veW7x3+F9KsWSPrfX2lfWvX\n7lpeW7u2XKQ/6KCRm+0OP7x8V5VPMIa6pHfYtAmefLIE/NNPl+A+6KBdlwMOaDyaH7Z5cwn3F14o\nJ4xt20auecDI+tBQeZ7R+vXvXDZsGCltbd8+sj5jRgn5Aw5457LPPuUENHyz3fPPl+Mcfni5eL5j\nRym37dix6zJrVjmJ1J+ohteHZ2mNLr9F7HqSrF8f/s7RJ6RZs3a/zJhR/sUzfPKtX84+Gy67bOy+\nNtQlTVuZ5UQwc2Zzo+9MeOONEu6vvVZG+X19JUCH1/v6ykmjvpRUvz7WBfJt28rJYbicVb/Mm1fa\nN/pkNLy+ffvIMlwy2769nAj22mukTFa/HH98KZONxVCXpAqZaKh77VySKsRQl6QKMdQlqUIMdUmq\nEENdkiqkYahHxOKIeCYiVkTEtbvZ5xu1z5+IiJOmvpmSpGaMG+oR0QfcBCwGjgUujohjRu1zLvDB\nzFwIfBH4ZovaWhkDAwOdbkLXsC9G2Bcj7IvJazRSXwSszMxVmTkILAXOG7XPZ4BvA2Tmg8C+EXHA\nlLe0QvwfdoR9McK+GGFfTF6jUD8YeLFue3XtvUb7HPLumyZJmqhGod7sLaCj73by1lFJ6oBxHxMQ\nEacCSzJzcW37OmBnZl5ft89/AwYyc2lt+xngk5m5dtR3GfSSNAkTeUzAzAafPwIsjIjDgJeAC4GL\nR+2zDLgaWFo7CawfHegTbZQkaXLGDfXMHIqIq4F7gD7glsxcHhFX1T6/OTN/GBHnRsRKYBNwectb\nLUkaU9ue0ihJar2W31HazM1LVRUR34qItRHxZN178yPivoj4TUTcGxH7drKN7RIRCyLiJxHxVET8\nOiL+qPZ+z/VHRMyOiAcj4vGIeDoivlp7v+f6YlhE9EXEYxHxg9p2T/ZFRKyKiF/V+uKh2nsT6ouW\nhnozNy9V3K2U//Z6fwrcl5kfAn5c2+4Fg8B/zMzjgFOBf1/7f6Hn+iMztwKfyswPAycAn4qI0+jB\nvqjzJeBpRmbO9WpfJNCfmSdl5qLaexPqi1aP1Ju5eamyMvNnwBuj3n77Zq3a6/ltbVSHZOYrmfl4\nbX0jsJxyj0Ov9sfm2uosyvWqN+jRvoiIQ4Bzgb9lZHp0T/ZFzehJJRPqi1aHejM3L/WaA+pmB60F\neu7u29psqpOAB+nR/oiIGRHxOOW/+SeZ+RQ92hfA14FrgJ117/VqXyTwo4h4JCKurL03ob5oNKXx\n3fIq7DgyM3tt/n5E7AV8H/hSZm6Iuh+i7KX+yMydwIcjYh/gnoj41KjPe6IvIuJfAusy87GI6B9r\nn17pi5qPZ+bLEbE/cF/tvp+3NdMXrR6prwEW1G0voIzWe9naiDgQICLeD6zrcHvaJiL2oAT6dzLz\njtrbPdsfAJn5JvC/gY/Qm33xMeAzEfE88F3g9yPiO/RmX5CZL9deXwX+gVLCnlBftDrU3755KSJm\nUW5eWtbiY3a7ZcBltfXLgDvG2bcyogzJbwGezswb6z7quf6IiP2GZzBExBzgLOAxerAvMvM/ZeaC\nzDwcuAj4P5n5B/RgX0TE3IiYV1vfEzgbeJIJ9kXL56lHxDnAjYzcvPTVlh6wi0TEd4FPAvtRamFf\nBu4EvgccCqwCLsjM9Z1qY7vUZnf8FPgVI2W564CH6LH+iIjjKRe8ZtSW72TmDRExnx7ri3oR8Ung\njzPzM73YFxFxOGV0DqU0fltmfnWifeHNR5JUIf6cnSRViKEuSRViqEtShRjqklQhhrokVYihLkkV\nYqhLUoUY6pJUIf8fTHuMBionqrEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc0d5703a90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Plot the cost history\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.plot(cost_history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "##Visualizing predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def PlotPredictions(test_samples, predictedLabels):\n",
    "    plt.figure(figsize=(3,2))\n",
    "    f, axarr = plt.subplots(2,5)\n",
    "    for i in range(2):\n",
    "        for j in range(5):\n",
    "            index = i*5+j\n",
    "            axarr[i,j].imshow(test_samples[index].reshape(28,28), cmap='Greys', interpolation='nearest')\n",
    "            axarr[i,j].axis('off')\n",
    "            axarr[i,j].set_title('%d' % predictedLabels[index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random examples\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc0e39da1d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWwAAADeCAYAAAAZ6jpzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHdtJREFUeJzt3Xu8VWMex/HPoxClTLnHIDKuKXKdSSMmptcg16KkmZFr\nk0suw4jcisGYIpdUlNsgKilh3C+RMMIQg4khlWvo4rbmj31+59l7nb1Pp3P23ms9+3zfr1cv2Wef\nvZ+zztq/futZv+f3uCiKEBGR9Fsl6QGIiEjdKGCLiARCAVtEJBAK2CIigVDAFhEJhAK2iEggFLBF\nRAJRUQHbOdfeObfMOXdr0mNJA+fcE865pc65r6v+vJn0mJLknFvNOTfWOfdf59xi59wrzrn9kx5X\n0pxzrZ1zk5xz31QdmyOTHlPSnHMDnXOzq+LJzUmPx1RUwAZGAbMArQbKiICToyhaq+rPNkkPKGFN\ngQ+AvaIoagmcB9ztnNs02WElbhSwDFgP6ANc75zbNtkhJe4j4GJgXNIDyVYxAds51xv4AngUcAkP\nJ010LKpEUbQkiqILoyj6oOr/pwHvAzslO7LkOOeaA4cAQ6qOz7PAFODoZEeWrCiKJkVRNAX4LOmx\nZKuIgO2cawlcCJyGAlTccOfcIufcM865rkkPJk2cc+sDWwFvJD2WBG0F/BBF0X+yHnsV2C6h8aRN\nquJJRQRsMpcuY6Io+hhNh2Q7G9gc2AgYDUx1zrVLdkjp4JxbFbgduCWKoreTHk+CWgCLY499DayV\nwFjSKFXxJPiA7ZzrCOwD/N0eSnA4qRJF0awoir6Nouj7KIomAM8CPZIeV9Kcc6sAt5KZtx2Y8HCS\n9g3QMvZYKzJBW1IWT5omPYAi6ApsBnzgnINMxtDEObdNFEWdkxyYpI/LnCRjgXWBHlEU/ZjwkJL2\nNtDUObdl1rTIjsDrCY4pTZRhF9looB2Zk6wjcAMwDdgvyUElzTnXyjm3n3OumXOuqXOuD9AFmJH0\n2BJ2PbA1cGAURcuTHkzSoij6FrgPuMg5t6Zz7lfAAWSuQBot51wT51wzMkltE+fc6s65JkmPK/iA\nHUXR0iiKFlb9WUDmEm9pFEWpurubgFXJzO0vBBYBJwMHxW4uNSpV5XvHkfnH/ZOs+vTGXnd8ErAG\nmXPlNuCEKIoadc0+MARYQuY+UF9gKfCXREcEOG1gICIShuAzbBGRxkIBW0QkEArYIiKBKGVZX2Oa\nHF+ZWs3Gclx0TPKr63HRMcmvsRyXvMdEGbaISCAUsEVEAqGALSISCAVsEZFAKGCLiARCAVtEJBCV\n0K1PGrGffvoJgCVLlgDw8MMPAzB58mQAbr/9dgDWXHNNALp37w7A7rvvXv0anTp1AqBbt24ArLKK\n8hhz662ZHlD9+vUDoKojJq1btwZg1KhRAPTq1SuB0RXX1KlTAejZsycArVq1AuDzzz9PbExxOjNF\nRAKhgC0iEohSdusr+Yqk448/HoAxY8bU+No999wDwCGHHFLqYUAgK7WWL8+0f7ZphPHjxwPw5Zdf\nAnDOOecA/rI3W4cOHQCYOXMmAGusscaK3q4sx+T7778H4LDDDgNg2rRp+d+g6jy3n61lS7/Jir3G\nddddB/hzpnnz5vUdVm1Su9Lxq6++qv77lClTALj66qsBmDNnTt7v+fnPfw7A+++/35C3TvTz8+GH\nHwKw1VZbAf58sHNlwIABgD8/ykQrHUVEQhZkhj169GgATjzxRMD/S7jWWn7f0EcffRSAnXbaqVTD\nyJbKDHvBggUA/P3vme0u7QbS/PnzV/q17NjOmzcP8DdkalHWY/LWW28BsPfeewOwaNEiAA4++GAA\nhg4dmvP87Ax7xozMJjx2Pu25554ATJ8+HYAWLVo0dHjZUpdhP/300wAMGjSo+rFCGXVc/LPXv39/\nwGfmdZTo58fO6S222KLW59lVxCabbFLsIeSjDFtEJGRBZdgff/wxAPvssw8Ab7/9NuD/ld9hhx2q\nn/vKK6/kfY2vv85sBn3vvfcCcMABBwCw9tprA9CkSb22bUtFhh3/2YYNGwbAu+++W+v3bbrppgDs\nuuuuAJx22mk1nrPeeusBsPnmm9d1OIkck+eeew6Ab7/9FvDnSm2lejaHaeePHccjjjgCgDvvvLNY\nw4MUZdjXX389AKeccgoAP/7Y8P2I7bOYna3/7W9/W+G3rcRbFP242Jy1lS7a/a+4K6+8EoBTTz21\n2EPIRxm2iEjIglg4Y3Oxli298847OV/v3LkzAJdffnnB17AriZNOOgnwWZM9bnOetiAgBN988w0A\nd911F+AzALvyiNtggw0AnyHY1YU9blcZIbP555VhVyB2PM1HH31UlDGljVU72JVUbZn1mWeeCcDA\ngQMBuPTSSwGYO3cuAE8++WTO8+3zZAtqAAYPHgxA27ZtGzz2Ulh11VUBf7+rUIadBsqwRUQCkeoM\n2+asb7zxRqBmZn3hhRcCcN55563wtaxSIj4faa/RrFmzhg22jCwTPOiggwB44okn8j5vv/32A+CC\nCy4A4Be/+AVQGZl0Q73++uvVf7crN2N16vbfShHPrH/44YeCzz3rrLMAuOiiiwCfhV577bWAn/f9\n5JNPgJoVFtmv/eCDDwJw7LHHNuwHKLFjjjkG8OsR0kgZtohIIFKZYds82C677AL4OWxjc9ann376\nCl/L7lBbFm71ogceeGDO42n3yCOPVP/dMoH4cbHmRTbP2LFjRwBWW221cgwxKNboB2qu7LSKktCb\nQFnToltuuQXwmWM8s954442B3OoHm7O2zNpYFZX9t02bNnnfO/vYZVdvpZmtLbCVrvfdd1/O1+04\nnnzyyUDNY1MOYZ+RIiKNSCoz7KVLlwI1M8jVV18d8NUg1jIzH2u3aXe5LYu6+eabgXDaQVo9+bnn\nnlv9WKHM2vpoKKNesddee22Fz7HeEqGxzLpr164A/Pvf/877PMusH3vsMWDFK/1WRnb2udtuuxXt\ndUvJ4kuhlYx2ziR5b0MZtohIIFKZYRdaGTV8+HDAZw5xllUD7L///nmfs+222zZwdOV19913A/Dy\nyy8XfI414C/hqtWK8emnnwK+f0g+tqrTKiJCYVVUdn+mUD2+eeCBB4CGZdZNm2ZCiK2AtX4b2bXd\nlvGHtMahNi+88AIAe+21V9nfWxm2iEggUpVhP/TQQ4CvG4777LPPADj00EMBfxe3trv5Nt9kd3hD\nuWNtbD4tuztedt9igKuuugrwfTSsrtiqAkKqMS+17777DoDFixcXfI6df6Edt8cffxyA//znP7U+\nz2qsrS6/Ieyqzo6rya5Esc9p2uuwja1vGDFiRN6v29oN6+Zoc9/loAxbRCQQCtgiIoFIVXvVo48+\nGqh7O8v4tk+1PadPnz4A3HDDDUDtJYH1UPL2kNk3Ha3R/uzZs2v9nl/96leAv0lrBf/rr79+fYaw\nslLRctbY1JgdA9sEo7bn2lSbbUV3/vnnA75UdI899gDg97//ffX3WiOpLbfcEsg7rVL09qrWmmDf\nfffNfGOBz3Sh5eYNYa1o4+0OsktLbcOMWloipOpcWbZsGQA9evQAfIOr+HlhW+tlb5xSRGqvKiIS\nsqAybNvw027AWemQ3WSJ3/iAmln4UUcdBfgMq0g3lsqaIVgGYO0/r7nmGsD/i29bgcWts846ANxx\nxx2Az7ytNKvIUpU12RZi22+//YoHU4crt+zn2XEFvzmxbVU2cuRIICcLK3qGHd8yL+7II48E/KKx\nYmTWdlPRbnhnL+yC3Btx2eW2BaTqXDF2JWL/jZ8XX3zxBaAMW0RE8khVWV+cbaBqGYKVq8U3gO3Q\noQNQeAluNssue/bsCfhGLyGxqwJb8GCtYy0DsJ/NFkZYZmWLRrp37w74trWhlFutDFsYM2HCBMBv\n8rCirBn8+RRfZGUlobawxpZcW7kp+GzSyjHzXfWVmzU4K2ZmbYvb4pm1ze9ecsklDX6vtLPP3ZAh\nQ8r2nsqwRUQCkao57PqyzCc7w7Ys0+7s3n///Xm/txgbj5LSOTj72W2uu0uXLoDfdLZ58+YA9O/f\nH/DzrUWS6DGxCpnnn38+8wZ1mJd+4403AJ8dF7mSyJR9DtsWWrVo0aKuL1lDPLMu1OTfHl/JDDuV\nnx+LGXYVHj+H7HG7Z1TkBTSawxYRCVmq57Dryv7Fy86erO7aaimPO+44wM9hNwY2n2gZ43vvvQf4\nTNuyz5tuugmAQYMGAb6GOERWZ/3SSy/V+rz27dtX/92aPFk71brMc6eJNVcqJsuobTs6y+LjmbVt\nZGD3U6w6phLYFnuF2JJ723B43XXXLfmYlGGLiASiIjLsfGxzUKuoGDx4MNC4Muw4y7gnT54M+CqH\nefPmAX5LtldffRXwde9pZvP0t912G+BXssZZ1ty3b18gdz43vglvaCzrLdQEbcqUKYCvtrL7NlbP\nn8/cuXOBFW8+YBUotgpUSksZtohIICo2w7Y7/o2JzTdaplWoysHm2oYNGwb4+X5rOfrBBx8AYWTY\nVvds21wVmn/eaKONAN9mt5JYb5hFixbl/Xq/fv0AePHFFwFYuHAh4GvT6+OMM84AClemNCZ2HG3j\n4lJShi0iEoiKyLCtPjK7ptzmMu0O7qOPPlrjOZVm1qxZgM9+rBrE/t+qRf773/8Cvr9GyKxb3Ip+\nr5dddlk5hpMIu19jFTKF5vGt50x92FWbVdfYOVWOyoi0mzNnTtneSxm2iEggKmKlo/0LZzXX4LOO\nbbbZBvB9imvbyqgBUrFSy+7+Z/ciBmjZsiUAnTt3BmDmzJkALF26NOd5bdu2BXyFgHWeq6eyHJMr\nrrgCKLzyzn5mqzlPgaKvdIyzTHvMmDFA/c5xuxdgtemWnZeooiYVn584+zwNGDAAgPHjxwM175P8\n4Q9/AGrvsV4PWukoIhKyisiwTfbqNqsfja//t/X+w4cPB/zqvgZKRYZgP+vEiRMB6N27d52+z/ph\nP/PMM4Cvx26gshwT23h4r732ynncMmu7d2F9U1Kg5Bm2sZ4x1s953LhxOV/v1KkTACeccEKN723X\nrh0A3bp1a+gw6iIVn58VsWqqeHWN9RzJvsIvAmXYIiIhU8AWEQlERU2JZC+1nT59OgCHH3444KdE\nrD1kkaZCTKou6ex3+tRTTwF+KbqV8XXs2BHwJVm2+KGBNxnjUnVMUqRsUyIB0blSk6ZERERCVlEZ\ndoKUIdSkY5KfMuyadK7UpAxbRCRkCtgiIoFQwBYRCYQCtohIIBSwRUQCUcoqERERKSJl2CIigVDA\nFhEJhAK2iEggFLBFRAKhgC0iEggFbBGRQChgi4gEQgFbRCQQCtgiIoFQwBYRCYQCtohIIBSwRUQC\noYAtIhIIBWwRkUAoYIuIBEIBW0QkEArYIiKBUMAWEQmEAraISCAUsEVEAqGALSISCAVsEZFAKGCL\niARCAVtEJBAK2CIigVDAFhEJhAK2iEggFLBFRAKhgC0iEggFbBGRQChgi4gEQgFbRCQQCtgiIoFQ\nwBYRCYQCtohIIBSwRUQCoYAtIhIIBWwRkUAoYIuIBEIBW0QkEArYIiKBUMAWEQmEAraISCAUsEVE\nAqGALSISCAVsEZFAKGCLiARCAVtEJBAK2CIigVDAFhEJREUEbOfcxs65qc65z5xz851z1zjnmiQ9\nrqTpuBTmnGvvnFvmnLs16bEkTedJTc65gc652VXnyM1Jj8dURMAGRgKfAhsCHYGuwEmJjigddFwK\nGwXMAqKkB5ICOk9q+gi4GBiX9ECyVUrA3g64K4qi76IoWgDMqHqssdNxycM51xv4AngUcAkPJw10\nnsREUTQpiqIpwGdJjyVbpQTsh4CjnHNrOOfaAr8FHkx4TGmg4xLjnGsJXAichoK10XlSWKrOkUoJ\n2EOB7YHFwIfAi1X/OjZ2Q9FxibsYGBNF0cdoOsQMRedJIak6R4IP2M45RyZDuAdYE1gHaO2cuzzR\ngSVMx6Um51xHYB/g7/ZQgsNJBZ0nK5SqcyT4gE3mBNsZuDaKou+jKPocuAXokeiokqfjUlNXYDPg\nA+fcfGAwcKhzbnaio0qWzpPaKcMusk+B+cCJzrkmzrm1gWOAV5MdVuJ0XGoaDbQDdiRTDXEDMA3Y\nL8lBJUznSR5Vx6IZ0BRo4pxbPQ2ljsEH7CiKIuAQ4AAyJ987wHIyN5UaLR2XmqIoWhpF0cKqPwuA\nb4ClURSlqhKgnHSeFDQEWAKcDfQFlgJ/SXREgMv8vkREJO2Cz7BFRBoLBWwRkUAoYIuIBKJpCV+7\nMU2Or0ytZmM5Ljom+dX1uOiY5NdYjkveY6IMW0QkEArYIiKBUMAWEQmEAraISCAUsEVEAqGALSIS\nCAVsEZFAKGCLiARCAVtEJBClXOmYqK+++gqAvn37AvDAAw8A8L///Q+Atm3bJjOwIli+fDkAkyZN\nAuDhhx8GYPz48TnPa926NQAvv/wyAJtssgkADz6Y2a6vS5cuALRo0aLEIw7Tc889B/jjZOfSmDFj\nAFh11VWTGdhKWrZsGQAdOnQAoF27dgBMnjwZgB9//BHwPyfAK6+8AkBmQxqwrp4XX3wxAGeeeSYA\nq6++eknHngaPPfYY4H//t912GwDdunUr+1iUYYuIBKKU/bDLtubfMoRHHnmk+rE+ffoA8OWXX+Y8\n99prrwXgxBNPLOYQytoL4ZxzzgHgiiuuyLxg1e/QsqG4n/3sZ4A/JnfddRcAO+64IwAXXHBBzvPt\n688//3z1Y/fccw/gs/Q6CL4/xODBgwEYMWIEAKutthrgr1i23nrr+rxs2XuJfP311wCsvfbaOY+f\ncMIJAOy9994A9OrVq86vedRRRwEwbtw4oMFXG6k8V7777jsA2rdvD8CHH34IwAYbbADAxx9/XMq3\nVy8REZGQVUSGfeeddwJ+jqk2LVu2BGDBggWAz5oaqOQZwvz586v/vtNOOwGwaNGizAtW/Q7XX399\nAHbeeef8b1z1vMcffxzwc+GFnmevBz6rtOyiDsqaNX300UeAzybrmf0CsHTpUgD23HNPAF577TUA\nNt10UwDefffder82CWTY3377LeDv29gxWmWVTL5m58PYsWNrfK+dS6NHjwbgzTffBOCnn34CYNSo\nUYDP1usplRm2zf2vueaaOY/37t0bgDvuuKOUb68MW0QkZEFWiViGYHdr7Y51XSxevBjwWfkxxxxT\n5NGVRvb8o2U9M2bMyHnOIYccAvh5+rgnn3wSgKeeeirn8SZNMptBWwY5YcIEIDfDXonMuqxsXtEq\nIGze8d577wVg//33r/Nr2RWHZVCWWYeuefPmgD8/rJrIsuT33nsPgJtvvrngawwcOBCA007L7M07\ncuRIAN56660SjDjdfv3rXyf23sqwRUQCEVSG/f333wOw/fbbA76mOp+//CWzI73NvVrtcajWWGON\n6r+fddZZQM0M2+qGrQa9VatWAMyePRvwdaPxapLzzjsPgCFDhhR72CVnFS5WDWTzslOnTgXql2Fb\nzX7cE088Ud9hpsKgQYMAf2Vq1VWnnHIK4K8soOa9Hfvs3X333TmPf/HFF6UZbAosXLgw7+MHHnhg\nmUfiKcMWEQlEEBm2VUNYJlhbZm3mzJkDFM6KbC7u7bffBnxGHr8jnEYdO3YEfDWHzUW++uqrgJ/D\ntvrrP/3pTznPs5/V5v4tEw/JCy+8AMDEiRMBn1nb1cPhhx9e79cuVM/+6aefAitVi54qdt5cf/31\nABx33HGAv69j9zgAfvOb3wD+noA995NPPsl5zSOPPLKEI07W9OnT8z4er2cvJ2XYIiKBCKIO2+YU\nDzrooLxft7u2tgIQ4B//+AdQ+53vbFbHW89qiLLWkVqmbPO0dve/UGZo/vznPwNw0UUXAT4rLZGS\nHpMePXoAvo+Knce777474Ps/rEyvC8s0rQdLnM2TN7D3SuK7plt9sV2BWRZtK18BbrrpJgCuuuoq\nwK9+NX/9618Bf6XawHMplXXYN9xwAwAnnXRSzuNLliwBoFmzZqV8e9Vhi4iELNVz2JYJWIewuB12\n2AHwGfi0adOqv2Z1uIXYXLVVRrRp06Zhgy0jy2bsisMy5nhPkDibsy5xZp0oqwqpTxc5q54opFK6\nGlpmOHToUADOPfdcwN8DAdh1113zfu/w4cMBOP7444HKPpfSSEdbRCQQqc6wbU7xX//6V96v20q0\nzTbbDPB38WtjK7Yuu+wyILe+OVRbbLFFnZ730ksvAcn08S02m7Mu9N/6sFWT8dew/jOVxq7QLMPO\nx1a7XnPNNQAcfPDBgDLrpOioi4gEQgFbRCQQqZ4SWW+99QBftH/dddflfV5dpkKMlXuFsr1TMV1y\nySWAbxta4rKkkrISxngp40MPPQTA2WefDdTt5qM1E7NStfhrZpeLVgJrXWDloLWxjSt++ctflnRM\naRRfJJQGyrBFRAKR6gzbWOZjC2SOOOKIvM+zjBx8KeA777wDwJVXXlnCESbDltXb8mC7EXTyyScD\nvo2q3Zy1RvXdu3cH/NLbEMvVrCTtmWeeAXyTfluybiV6v/3tbwGYN28eANttt12N17KyyEK23Xbb\nhg84BWzjBVt2bsekNiVcWJd6N954Y9JDqEEZtohIIILIsK30zrKluHXWWQeAWbNmVT9mDXosOze2\npLlp0yB+9Lys1eWwYcOAmo2PTj/9dMCXa2288cY5z7M2rNYY63e/+10ZRl1ctrDDrpzimyrbwg4r\nybN52+yrMGPNxQrZcMMNGzbYhNnCMpuztraqxq4gsudsP//8cwDOOOMMIHdD5sbK7v0kGTuUYYuI\nBCKINNOarRx22GE5j1tR/8yZM4H8bS/jGXYlsE1i40up7fhstNFGgM+oLQsdPHhwzvNtezRr7LPv\nvvuWaMSlYxVEu+yyC+CPgW1abJm1zcVaU/rsrNnaZVpWGde+fftiD7ss7Gc/+uijgZqZtTU6sysu\nqyICf87Y8bJ7BLbdWCWzeGNNsYydY8qwRURkhYLIsG+//XbA19ia+++/H/Cbx9ZFv379ijewhMS3\naTKWCca3dzr11FMBv9T/0ksvBXwGNmDAAMBvqFqfxklJ69SpE+ArIexnsZaocdmtRKdMmQJAnz59\nAH8vwBpJhdq+4M033wT8793Yz/Piiy8CsNZaawH+agV8hm2VJFZtZZsgVDLbVjCN258pwxYRCUSq\nM2ybU7z88stzHre2qltttVXB73366acB/69kr169gMqoqbUMIM425y3k/PPPz/l/qz+2pkd9+/YF\n/Oq2kG299dYNfg27cqu0VbHHHnss4O91mLZt21b/3erV33jjDQDGjRsHwMiRI8sxxFTq3Llz0kNQ\nhi0iEopUZ9hXX301AO+//z7ge1/YZqHxtpfZPUVsJZyxtqor2kYrBPFWorZlWF398Y9/BGDs2LGA\n39T4vvvuA/wKytquYCT9rO2wbdZh1Q9WWx2X3Vtm3XXXzfmabSZi51wlfI7qyn7mlblXVirKsEVE\nApHqDDuuSZMmALRq1Srn8R9++AGACRMmVD9mq/gqkW0FZhulWr21ze1bFYjN11sFhbG5SpuntA2I\nzaRJkwDf8a7SxTfyNdZzJVRWZ229YizDtvs5VnVlX7crK4A5c+bkvNaMGTMAv8o2XolUyexqIt86\nj3JThi0iEoigMuzly5cD/o613b2fOHEi4HsmZLO+GnvssUc5hlgWVjdrvVWs657NRdsKRqun7tq1\nK1Bz/lH9ITLsnki8x3aHDh0SG1MxWR2+9Zax37tdgdkVa/aVltXom549ewKNK7M29jlKQ7WQMmwR\nkUCkOsO2u9zG5qptZV5tLLO2zXYr6a629XOYOnUq4Oupbe7aekbYFUl8jrbQsWjTpg3ge080Ftbp\nr1Ln7O2zYL/fQYMGAT6jjt/DyGYVJr179y7lEFPFOjpaRt2lSxegZt16EpRhi4gEQgFbRCQQroRb\nADX4hW0KxG4mHnrooXmfZwX/I0aMqH6sf//+QNlaIa7MfEvJDriVXtkNI9u0+Nlnn828cYEpEbtU\nnj17NlC08qVUHJO6sKkjm1qyVgjW8GjzzTcv5tvV9biU7JhYy9R//vOfAMydOxeAadOmVT/HWona\n52i33XYr1XAgpeeKTafaxt02tVgmeY+JMmwRkUCkOsMOSCozhITpmOSXeIadQjpXalKGLSISMgVs\nEZFAKGCLiARCAVtEJBAK2CIigShllYiIiBSRMmwRkUAoYIuIBEIBW0QkEArYIiKBUMAWEQmEAraI\nSCAUsEVEAqGALSISCAVsEZFAKGCLiARCAVtEJBAK2CIigVDAFhEJhAK2iEggFLBFRAKhgC0iEggF\nbBGRQChgi4gEQgFbRCQQCtgiIoH4PyxnClFIwoO9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc0ceef45d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predictedLabels = get_preds(x_test).argmax(axis=1)\n",
    "\n",
    "indices = np.random.choice(y_test.shape[0], replace=False, size=10)\n",
    "\n",
    "PlotPredictions( x_test[indices], predictedLabels[indices])\n",
    "\n",
    "print \"Random examples\"\n",
    "\n",
    "plt.savefig(\"randompreds.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Incorrect predictions\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc0ce817dd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWwAAADeCAYAAAAZ6jpzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH/xJREFUeJzt3Xmc3dP9x/HXISGI2EOstdQSxBJbI7TVUPwopXaJtahS\npShFf/KLfV+K1BaVWkJR+74vJahdiDXEvsSSSJD4/v64855z585MMpm5936/Z+b9fDw8ZO7cuXPm\nO9975nPO+ZzPCVmWYWZmxTdL3g0wM7O2cYdtZpYId9hmZolwh21mlgh32GZmiXCHbWaWCHfYZmaJ\n6BQddghh/hDCDSGEiSGEt0MIO+XdpiIIIfwzhPBBCOGrEMKbIYSj8m5TnkIIs4UQLmm4R74KITwT\nQtg073YVQQhhxxDCmIb30OshhIF5t6kIQgg/DiFMCSGMzLst0Ek6bOA8YArQG9gFuCCE0DffJhXC\nicDSWZb1AjYDDuziHVQ34B1gw4ZrcjRwTQhhqXybla8QwsbAScBuWZb1BDYA3sy3VYVxHjAaKMQO\nw+Q77BDCXMA2wDFZln2TZdmjwI3A4Hxblr8sy17KsmxK2UNTgY/zak/eGu6PoVmWvdPw8a3AW8Ca\n+bYsd0OBoVmWjQbIsuyDLMvez7lNuQsh7AhMAO4FQs7NATpBhw0sD0zNsuz1sseeA1bOqT2FEkI4\nP4QwCXgJOC7Lsv/m3aaiCCEsTOn+eSnvtuQlhDAr0B/oHUJ4LYTwbgjh3BBCj7zblqcQQi9Kf8gO\npiCdNXSODrsn8FXFY18Dc+fQlsLJsmx/StdoEHBcCGGdnJtUCCGE7sAVwGVZlo3Nuz05WhjoDmwL\nDARWB9agNF3UlQ0DLm4YaRRiOgQ6R4c9EehV8dg8lDptA7KSB4BrgS6/IBtCmAUYSWnd44Ccm5O3\nyQ3/PzfLso+yLPsMOAPYPMc25SqEsDrwC+AsPZRjc5rolncDqmAs0C2EsFzZtMhqwIs5tqmougOf\n5d2IPIUQAnAJsBCweZZl03JuUq6yLJsQQhifdzsK5qfAj4B3SrcLPYFZQwgrZVm2Vp4NC52hvGoI\n4SpKw5a9KS0g3QL8JMuyMbk2LEchhIUoRQk3U4okBwHXAIOyLHsyz7blKYQwnNIf9EFZlk3Kuz1F\nEEIYSimL6H8oLUzfBNyXZdn/5tqwnIQQ5iBOqQbgUEod+H4NI5DcdIYIG2B/4FJKGRCfUrqwXbaz\nbpAB+wEXULrpxgKDu3hnvRSwD6U/YB82RE8A+2RZdlVuDcvfMGBBSvfIFGAUcHyuLcpRlmWTiVNF\nhBAmApPz7qyhk0TYZmZdQWdYdDQz6xLcYZuZJcIdtplZImq56NiVJsdnJk+zq1wXX5OWtfW6+Jq0\nrKtclxaviSNsM7NEuMM2M0uEO2wzs0S4wzYzS0Rn2enY6bz5Zql+/LLLLgvALLPEv63TpnXp8hdW\nAN9//z0A77zzDgAPP/wwAHPMMQcAa60VS2707t0bgLnndgHNjnKEbWaWiKQi7IkTJwJw9dVXA/DH\nP/4RgMmTS9v+11knlno+55xzAFh77bXr2cSqOf3004EYWZdH2NZx5ddz7733BkBlGvr16wfAgQce\nWP+GFdzUqVMBGD58OBDfg7p2ZfVZGi2yyCIAbLLJJgCMGDGi5u3srNwLmJklwh22mVkialmtr2ov\n/PnnnwNxiDpq1KgZfs2cc84JwN133w3AuuuuW63mtKRmO7U0dC8far722msALLPMMjPzUvVW6N1r\ns846a+O/dW0rh/WPPvooUPV7J8mdjs8++ywARxxxBBDfV7L77rsD8dptscUWjZ9bYoklmjy3f//+\nlS9fyHtlhx12AOCaa64BYKWVVgLg5Zdfrse3905HM7OUFXLRUZHOQw89BMBvf/tbAN544w2g5YWN\nSpMmlQ4Tufzyy4GaR9g1c9JJJwFw1FFHNT628cYbA3DPPfcAsPTSS9e/YYm7+eabmz221157AfDx\nxx8DcOSRRwJw33331a9hBfP+++8DsOmmmwLwySefALDRRhsBcMUVVwAxda8zUMLCv/71LyD2N8sv\nv3xubRJH2GZmiShkhK2oWBGPrLrqqgAcffTRACy++OJNPr/LLrs0/vvtt98G4M477wTg22+/BWD2\n2WevfoNr6De/+Q0A48aNa3zsggsuAGC55ZYDkpnTLpTNN29+KLjuN41qurJPP/0UgLPPPhuIKbXn\nnXceEEe95WsBqZsyZQoAt99+OxBH+r169QKKcV84wjYzS0ShIuwXXngBgH322afJ45p/1pyttr9W\nWnDBBRv/rQhb/1dmyTzzzANA3759Afjxj39chZbXjqJmRTYASy21FBDntT2nXV2KrHRduwptN4d4\nb1144YVAHH3st99+9W9YnWjkqlG5XHzxxQCsuOKKQFzTeOqppwA4/PDD69VER9hmZqkoRISt7a4j\nR45s8rFoS2trkfV3330HxHk2iFGS7LHHHi1+rfJH9dd1ttlmm5mm50J/0fX/zTbbDIhz2rpeGlVo\nDs6m79JLLwViVoDWD7qKgw8+uPHfiioPPfRQAIYOHZpLm+pp/PjxTT4+66yzANhmm22AOAI55JBD\nAHjxxRcB2G677YD6jGwdYZuZJaIQEfZzzz0HwBlnnAHATjvtBMQiT8qCqKRIXDsgX3311cbPKUqa\nUc72ZZddBsTI+txzzwWgW7dCXJo2USQ9ZswYAAYMGADAzjvvDMTsGV1Xa9mee+4JwIknnggUf32j\no/T+0S7GK6+8stlzNDLVzuHOSBlklSNSvW+021iff/755wHo0aMHUN/MM0fYZmaJKEQYqV1Smk9W\n7qcibP1/rrnmAuJfez2u3Vct+fnPfw7AddddB8RoQiU1tXvyoosuavJ1mtNOgSICZdNoRKISrYoU\n3n33XSDOzXblvO1vvvmm8d+63zSHveaaa+bSpnrRXOwjjzwCwKBBg1p9rnY0Vs7rn3DCCUDnOJRA\nWSHa1ak56QUWWKDJ83RYg6jfWnTRRWvdxEaOsM3MElGIan1ffPEFECPmwYMHAzFnuvEFp1MkvZKi\nhuuvvx5oPgenCoCqHKboU8XWK1eMZ6CQ1cbklFNOAWJu7ZJLLgnUPG+7UNfkyy+/BOJ9oBEXxBGI\nPqfj2RZaaKFaNCX3an377rsvEDNBptuIVt5z66+/PhB3BWr020653itas9D7Q9dFaxoycOBAAB57\n7DEgvo+016PKXK3PzCxlhZjDnnfeeYG4i+qHH34A4l8+VU1rjf7S7bbbbo2PKUe5tdzt+eefH4iV\nuX79618DcT68fMVc2RapmlHedleoRaJaxhrFKZcWYvSovNsaRda50Zz1gw8+CMTdi5VRs/L3AXbc\ncUcgritpflePa/5bFe3K33upq1zD0HrHW2+91eTx8tpF9eII28wsEe6wzcwSUYgpkUpKVD/ssMOA\neETP008/3eR5mio57rjj2v29dKq6Fle0mUCLlZD+lEilv/71r0BcdOzMxaOeeOIJAD788EMgToXo\nY4hD4MpyvqnThhBNgeiEc20S22qrrQA49thjgVjcCJqeKg9xmmiFFVYA4vFpNVpwq6sJEyY0+VjF\n1eSmm24C4IMPPmjyeB4pjY6wzcwSUcgIu1LlQbQqxlKNgjTvvfdek9eW8rS+VA8/aM1PfvITIC5G\n6fpqEVJpX+eff34OresYHe+lqEgRtX6/WkQrX1jShhmlc2lTVWomT54MwD//+U8gjjwrU1T1+9WC\ne1voXtG1EqXgpkg/k445a40OVCkCR9hmZolIIsKupC2h1TieqKXDWCFutACYNm1ah79PkVUe9Kut\nuuVpTEWf19bvSz/DJZdcAsR5Wh2OscMOOzR5HOLRczo4Q4etbrjhhrVudlXp99jamk6/fv2AWGRt\nZlRGmdqyXjnfmxJt2NO9s/LKKwNxE5DS+VTsqdJqq61W6yY24wjbzCwRSUbY1dTaFvRVVlml8d+d\nubQkwAYbbADEDUuKrK+99trG59TzGKT20MaNW265BYglBjRPq0I++l2WH4elDVtbb701AMOGDQPi\nmoWKahWdDpEVZQPdddddADz++ONA/L22pdyusmyUsSWa9035EF4VTVO2x0svvQTEiFvXSZuGRKWX\nNTKrJ0fYZmaJ6LIR9sMPPwzAiBEjgOZZIpWFX7oCZYso0m5Lka28aZ5RWSGam77hhhum+3Xdu3dv\n9tgSSywBxLzlPn36ADB69Ogmny867SnYYostANh///0BWHjhhYF47NeWW24JQM+ePYGmo44DDjgA\niNdC5SO0FqB1pJRpBKVCVtp7oX0Jrc1d6yi1xRdfvNZNbMYRtplZIpKIsLWjUfnXw4cPB2Kk0JYM\nBkWNOkZLhdj1uKJLzX2qQFJnpjKiijAUUWvlP4VDaB944AEgtr2au1I196vrVPQIe9y4cUC8Fpp/\n1g5GFSu6//77gXiknvK3tRMS4JlnngHie0uFo4p+DdpDJZYVYVdG1iogp+v0n//8B4jZY/Wcx3eE\nbWaWiCQi7GWXXRZofrCu8ia1I091DiDO473yyitAnNNUwXWp3EWpXNbOTBGjymnqZ9e1uPfee4Hi\n515DzP7QnOrJJ58MxJK77cnwUGS63nrrAfG6FD0fXxkxyvLR4dSi0aSyHtZZZ51WX2v77bcHYsZM\nZ4ysRTs/daiJ3h/axaljBJUpowMMtEta91o9OMI2M0tEIY4Ia6ttt90WgBtvvHHG37yNx4kpB1PR\nya677tqmr6tQ1yOONIemI4sUKSvvtpLmH7ULsPLaaM66ypF1Xa/JqFGjADjttNOAOD975plnAm2r\nxKdaIqo/MmnSJCAe0qvMiQ6q+RFhd9xxBxCzQzS33fjCFb//H/3oRwAcccQRjc/R+sV8883X3mbM\njEIdJ1dJow0d1qCaO2PHjq3lt/URYWZmKUsqwlbVvD322AOIdbJb/OYziLCVYaIdfDoyrJ3qGiF8\n9dVXQMx4qYygKzNfKj9WDrqujeZ/qzxnnUvUpGp9AwYMAOKK//R2qyo7Qs/V7teRI0cCVd/pWLdD\neJXVoN2flfXklf0wo+P06qDQEbZ+/08++SQABx10EBBHbzXiCNvMLGVJRdiiSFunXihCKj8BQnUT\nlEmiFd8hQ4YAMbtAdQE6qNARQk5yvSY6WHixxRYDmkfYisQhzkkqYlKNjRrVkKlbhJ2QQr5/VJul\nb9++QDxdR/naqj1TI46wzcxS5g7bzCwRSU6JFFAhh3Q58zVpmadEmivkvaLFfS3GK9VRaZ49evSo\n5bf3lIiZWcocYVdHISOEnPmatMwRdnO+V5pzhG1mljJ32GZmiXCHbWaWCHfYZmaJcIdtZpaIWmaJ\nmJlZFTnCNjNLhDtsM7NEuMM2M0uEO2wzs0S4wzYzS4Q7bDOzRLjDNjNLhDtsM7NEuMM2M0uEO2wz\ns0S4wzYzS4Q7bDOzRLjDNjNLhDtsM7NEuMM2M0uEO2wzs0S4wzYzS4Q7bDOzRLjDNjNLhDtsM7NE\nuMM2M0uEO2wzs0S4wzYzS4Q7bDOzRLjDNjNLhDtsM7NEuMM2M0uEO2wzs0S4wzYzS4Q7bDOzRLjD\nNjNLhDtsM7NEuMM2M0uEO2wzs0S4wzYzS4Q7bDOzRLjDNjNLhDtsM7NEuMM2M0uEO2wzs0S4wzYz\nS4Q7bDOzRLjDNjNLhDtsM7NEuMM2M0uEO2wzs0S4wzYzS4Q7bDOzRLjDNjNLhDtsM7NEdIoOO4Qw\nfwjhhhDCxBDC2yGEnfJuUxGEEB4IIUwOIXzd8N+YvNuUp4b74+uy/6aGEM7Ju1158jVpXQhhxxDC\nmIZr9HoIYWDebeqWdwOq5DxgCtAbWAO4NYTwXJZlL+fbrNxlwO+zLLs074YUQZZlPfXvEMJcwIfA\nNfm1KH++Ji0LIWwMnARsn2XZ6BBCHyDk3Kz0I+yGm2wb4Jgsy77JsuxR4EZgcL4tK4zcb7KC+g3w\nUZZlj+TdkALxNYmGAkOzLBsNkGXZB1mWvZ9zm9LvsIHlgalZlr1e9thzwMo5tadoTgwhfBJCeCSE\n8NO8G1MguwGX592IgvE1AUIIswL9gd4hhNdCCO+GEM4NIfTIu22docPuCXxV8djXwNw5tKVo/gws\nDSwKXAjcHEJYJt8m5S+EsBSwIfCPvNtSFL4mTSwMdAe2BQYCq1Oaaj06z0ZB5+iwJwK9Kh6bh1Kn\n3aVlWTY6y7JJWZZ9n2XZ5cCjwOZ5t6sABgMPZ1k2Lu+GFIivSTS54f/nZln2UZZlnwFnUID3Tmfo\nsMcC3UIIy5U9thrwYk7tseIbgiPJSr4mDbIsmwCMz7sdLUm+w86ybBJwPfB/IYQ5G1JvtgRG5tuy\nfIUQ5gkh/DKE0COE0C2EsAuwAXBH3m3LUwhhAKUpomvzbktR+Jq0aARwYAhhoRDCfMDBwM05t6nT\npPXtD1wKfAx8CuyXZVmXzjmmNAc3DFgRmAaMAbaqWJztioYA1zX8obcSX5PmhgELUhrBTwFGAcfn\n2iIgZFmWdxvMzKwNkp8SMTPrKtxhm5klwh22mVkiarno2JUmx2dm+3dXuS6+Ji1r63XxNWlZV7ku\nLV4TR9hmZolwh21mlohOkYf9/vulIlprrbVW42N33FHaH9KvX79c2mRmVm2OsM3MEpF0hP3RRx8B\nMHz4cAA+/vjjxs9ttdVWAIwYMQKAn/3sZ/VtnJlZlTnCNjNLRNIR9kUXXQTA8ceXtviHEDNhFG2X\nP9ZZffPNNwCccMIJAFx6aelEsHXWWafJ8776qlQ2/L777qtj68ysWhxhm5klwh22mVkikpoSUWXB\nyZNLB0LcddddrT63Z8/SYdCLLrpo7RuWE037DBw4EIA33ngDiNNAN998c5OP55xzTgAuueQSAPba\na6/6NdYsce+88w4QpxrXX399AK644goAevSo/ZGPjrDNzBJRy3rYVX/h5557DoD+/fs3/UYNP0P5\nAuOqq64KwDPPPFPtZrSkrrUQtMh40EEHAXGRcemllwZgyy23bPLxwQcfDMBPf1o6NL1Oi46drj7E\npEml+v4bbLABEDdqXXjhhTPzMoWrJaL76bbbbmt8bOjQoQC89NJLACyzTOns5n322QeAAw44AIij\ntg4q5L0ybdo0AA455BAALr+8dKC8Fu9F76/TTjutmt/etUTMzFKWxBz2/fffD8Cee+7Z5q/58ssv\ngRiVr7baatVvWE7OPvtsIEbWGlk8//zzQIx6zjnnnCafP/TQQwHYZZddgJgW+cADDwAxYuzdu3dN\n21803377LQCff/45AH369GnxeZrDfPbZZ4GZux+LRCOFW265BYA//OEPAHz66afNnrvEEksAcMMN\nNwDwySefADB+fOmM2uWXX762jc3BK6+8AsCOO+4IwAsvvAA0H8kvu+yyQIyw68ERtplZIgo9h622\n6S/eoEGDgLglvfJ509skc/HFFwMwePBgAGadddaONq9czefgNGIAWGmllQD48MMPgRhxH3jggU2+\nZs011wRi5PzUU08BcdRRed2UVbL55pu3p4mVCjkvKS+//HLjvzXiUKStkUq3bqUBqCJvzWVOmDAB\ngOuuu67J89qo7nPYmosdOXIkEDdYKatI7d90000bv0brHJrfXnLJJYHmc9hzzz13NZpYiHvl1FNP\nBeCMM84Ampa6gPh+2XbbbYH4vlMmmu6T+eefvxrN8Ry2mVnKkpjDVsZHRyjn+MUXXwTg8MMPB9KZ\nry1fmW7rtntF0ooYFSHsvffeQBx16HXuvvtuIBbKqlIGQKFMmTIFgM0226zxsXfffReAbbbZpslz\ndb3+/ve/AzBq1CgAbr/9dmCmI+u6mzhxIhDn2jUiEK1pHHvssQDMMcccjZ/TdXrzzTcBuOyyywA4\n+uijgRiB65pUKarMhTJhzjzzTKB5ZK2+YsiQIQCssMIKANx7771AnMPW2sbqq6/e7HtoZK+1NK0N\nzCxH2GZmiSjkHLbmWk888UQA/v3vf0/3+T/88AMAs8zS+t+f1p5z7bXXAs2jq5lU8zk4RTYQr4t+\nd5qr1hykskc0x73IIosAMQLQqEJZJIoQ9HpHHXUUAMOGDWtPU6UQ85KiiLFv374AvP32242f05yk\nokXdI3feeScQo3HN7Su7op3qNoet36/y9TWS2m233YA4wpqZ9Rxlh2y33XYALLfcckCcH2+nXO6V\nyoNPKtfGtHPxoYceAmJUrBz1f/zjH0Dced0WmvP/4IMPgKajmgqewzYzS1khI2zN91x11VXTfZ7+\nMioXVHNMEA81UHTeWibJwgsvDMB7773X3uZCHSIE5YICrLHGGqUXqviZKj/WjkdFWpXzZt9//z0A\nAwYMAOC///0vAL/61a+AmHvbToWIsHVUnKJK5RH36tWr8Tn6ubWbT5GXokdlWShbSTtI26luEbay\nFhRhX3nllUCMGFULoz2++OILII7uNAJsZ256Xe8V5aErUj799NPb9o3bkI3WVhtuuCEQ95i0wBG2\nmVnKCrXM/d133wEx93NGtHrb0vyz/oL97ne/A2KkVTlPpY+1Cl6jPO0O064qiD+bdiiKaoVoJfrI\nI48EWs+E6d69OxDn1RRB3HTTTVVqdf0petIcoUYLU6dObfK88qwbVTt8+umngZhNoXlvXccORtZ1\np8wNrW1o5LDeeut1+LXnnXdeIM5lv/rqqx1+zXpRLRhlUc2s2WefHYhrGrvuumuTz7/++usA/PnP\nf272tYrOtZdiZjnCNjNLRKHmsFurxteaa665Bph+hsdrr70GwFxzzQXEymrHHXdck+fpOijDQnOe\nbZRLtb7PPvusyePtze1UfWztYlMUUBmVzqS6XBO1UZlFynh5/PHHS42omG/ceeedgbhrFmKev0ZV\n6667LhAzSqpc8bHuOx032WQTAH75y18C8Kc//alaL90sijz55JPb8zI1v1eUaw+w9tprA3E9o61U\ne0f3x8orr9zi88aNGwfENZFy88wzDxDXF6bDc9hmZilzh21mlohCTYloaFpemKecivSokHh7tPY9\ntLFGx/+UHz+mYcx0FCKFrb20FVcbbDSNUF5M6qyzzprZl63LNdl///2BmMYp2pCghemNNtoIiMPh\nlo5z0iKuUtMee+wxIF6XKqnblIgKhmmBUNNEGtJXg66NNhm1s4xxze4VlcQtb1d5IbXp0eKiNpxp\nS3prlOqoNNkxY8Y0e87uu+8OwIgRI2b07T0lYmaWskKl9Smyay0xvRoJ6zowUyk52r6t7cjV+B6p\nUdpf5fVX0agi08KgRorbb789AMcccwwAq6yyygxfQxtntHitDUfaVJUqLbirSNViiy1WtddWiqAW\nuot6QIg2xZSncc7oPa6faezYsQDMNtts032+FhBVNE0pjuVlMFTOQAvA7eUI28wsEYWKsGdE5QyV\nlnT88ccDM3e8fL9+/YA4h1ReZhNiepjmvqA65V1TULmeUX4NlEpYtJKr2hilIvJK1yvfet6S8i3B\nGm3pPlLKZ2cZbS244IIALL744lV7Tb1/tNGsaLRxSgWu2kL3klITZxRZqxzrSSedBMQj1jQyKz86\nTPdlRzfkOcI2M0tEoSLsPfbYA4DDDjusxc9XHomlBHXNn5UXDle0WFm8Xwdr6uBRiyrnsHUsFBQv\nshZFLG0tj6uNNuedd17jYzoaTEektXcDUtE8+eSTACy11FJVe01t1jr//POBuJ2/aFSUSuUFWqIM\nl1tvvRWI28WVHVJp9OjRQIycNRrXPaXX00zAiiuu2P4foBWOsM3MElGoCFsRdFtzw1X+U8V62nKA\ngVa3tQJc+Xm9Rg3z0wtLRY50QILySiGWYlXBqNSoROp+++0HwPXXX9/4OR0fp6JanYUOaVBRsI7Q\n+0GjXx36UOUc9apRUbeW1iGUm69DlRdaaKEmn1fErD5CEbi24Ve+pq7BPffcA9QmshZH2GZmiShU\nhP373/8eiLnSirhnpC051HqOCrhUPleHIKj05HzzzdfWZteddiaqDKoK0+tg1PbSAcW6NtMbsaRC\n0ZLyX7WbsXzX2t/+9jdgxlkBqVHZ3K+//rrDr6XdgbfddhvQ9ECN1FRG1ioCpUwo7aTW4cSVlM+u\nXYvKLunTp09N2lsu/XekmVkXUagIW1kJ+iuuEqe1+Guu45EUXSl3srVi/0WiVWpF1oqEH3zwQSCW\ndazMdlCUpF1fqr+hnFqNaBRhl0ehqc1dK/NDOyHfeustIGYPaIW/M9PBBTpouiN0UMHVV18NNJ/3\nLZrpHeelGjk9e/YEYn9TebiJaI5aax1aB1Hufz05wjYzS0ShImzRbkTVeNDctg4faAv9VdSctP7i\nquLfL37xC6BNlfgKRz+Doh1FEapIp48rsx50iKzmwCujEP1fUajm+1Ok2g2KrPfdd18g7k6zttF7\nT/PgqpdRdMo9nzhxIgATJkxo/NyMRhzaGar/K6+6CBkxjrDNzBJRqHrYCculHrbqJGjVW9GEIuXK\nCLq1jxU5KK+9SvWSC1EjXKOQrbfeGpi5ujM1Urd62DpyT0e/PfHEE23+Wo1uNTJR7RXN+1ZZze4V\n1RTRmhU03V8AMb9aazZ6bs7z9K6HbWaWMkfY1ZFrNKk8UuVhK+KuzPpQBoxWu/XxkCFDgKrP5+dy\nTVRhULVFVNlxp512qta36Ki6H8KrSFE79nSqUqXyzJktttgCiKfutHSgbBUVYjRWMI6wzcxS5g7b\nzCwRnhKpDg/pmqvrNdFmIB3Kq40z2hy0wAILdPRbVEvdp0S0oUpFvU499VQgbsU+5ZRTgHikGMTN\nVP37969WM6bH75/mPCViZpayQm6cMZtZKhim1Ky//OUvQMePZOoMtIFq/PjxQNx4pVIFJ5xwAhC3\nn0PnK4TVWTjCNjNLhOewq8NzcM35mrSs7nPYCfC90pznsM3MUuYO28wsEe6wzcwSUcs5bDMzqyJH\n2GZmiXCHbWaWCHfYZmaJcIdtZpYId9hmZolwh21mlgh32GZmiXCHbWaWCHfYZmaJcIdtZpYId9hm\nZolwh21mlgh32GZmiXCHbWaWCHfYZmaJcIdtZpYId9hmZolwh21mlgh32GZmiXCHbWaWiP8Hq2QK\nWw2TlVUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc0ce7ff7d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "incorrectPreds = np.nonzero(predictedLabels != y_test)[0]\n",
    "someIncorrectPreds = np.random.choice(incorrectPreds, replace=False, size=10)\n",
    "\n",
    "PlotPredictions(x_test[someIncorrectPreds], predictedLabels[someIncorrectPreds])\n",
    "print \"Incorrect predictions\"\n",
    "\n",
    "plt.savefig(\"errors.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc0ce1e5e50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQ8AAAD7CAYAAAB0WxGFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAABvRJREFUeJzt3d9rzv8fx/Hr+lIjLJFCimbXge1kSq2sZiK/HXHgQC3H\nzsSBYwdKQuRPQDuzFUeUQig/slq0lRxYbCWF1rjG9T3fvievx1Xfzz5Xt9vh6uG5g2v33qV3V7XR\naFQASv3nn/4FgH8n8QAi4gFExAOIiAcQEQ8gsrTZf2Bubq74/3onJiaiW2NjY8WbgYGB6NaqVauK\nN21tbdXoGIvShw8fij/b69ati259+fKleLNhw4bo1uvXr4s3/f39Cz7bnjyAiHgAEfEAIuIBRMQD\niIgHEBEPICIeQEQ8gIh4ABHxACJNv9ty69at4s2OHTuiW7VarXjz/fv36NaDBw+KNydOnIhusThN\nTU0Vbzo6OqJbbW1txZvx8fHo1tKlTf/ZVyoVTx5ASDyAiHgAEfEAIuIBRMQDiIgHEBEPICIeQEQ8\ngIh4ABHxACJNvyHT3d1dvEleAqpUsi+5+fXrV3RrZGSkeOPFuNaSvBg3PDwc3Wpvby/epH9Hq1ev\njnbzefIAIuIBRMQDiIgHEBEPICIeQEQ8gIh4ABHxACLiAUTEA4iIBxARDyDS9Fu1yVfXdXZ2Rrca\njUbxZmZmJrq1cuXKaEfrePz4cfHmwIED0a2enp7izdu3b6Nbz58/L950dXUt+JknDyAiHkBEPICI\neAAR8QAi4gFExAOIiAcQEQ8gIh5ARDyAiHgAkaZfjNu+fXvxpr+/P7o1NDRUvFm/fn1068KFC9GO\n1nHlypXiTfKVqJVKpfLnz5/izezs7P/t1v/iyQOIiAcQEQ8gIh5ARDyAiHgAEfEAIuIBRMQDiIgH\nEBEPICIeQKSafAsbgCcPICIeQEQ8gIh4ABHxACLiAUTEA4iIBxARDyAiHkBEPICIeACRpr8x7ubN\nm8Vv1l29ejW61dvbW7zZsmVLdOvQoUPFm76+vmp0jEWpq6ur+LM9NzcX3bp48WLxpqOjI7o1Ojpa\nvBkcHFzw2fbkAUTEA4iIBxARDyAiHkBEPICIeAAR8QAi4gFExAOIiAcQafrdloMHDxZv3r9/H916\n+PBh8eb27dvRrXq9Xrzp6+uLbrE47dq1q3hz+fLl6Nb09HTxZnx8PLp179694s3g4OCCn3nyACLi\nAUTEA4iIBxARDyAiHkBEPICIeAAR8QAi4gFExAOIiAcQEQ8g0vRbtWvXri3e3LhxI7p16dKl4s2b\nN2+iW7VaLdrROs6dO1e8WbFiRXTr48ePxZuzZ89Gtzo7O6PdfJ48gIh4ABHxACLiAUTEA4iIBxAR\nDyAiHkBEPICIeAAR8QAi4gFEmn4xLnkR6Pfv39GtU6dOFW96e3ujW+nvSOtYtmxZ8eb+/fvRrdOn\nTxdvlixZEt3atGlTtJvPkwcQEQ8gIh5ARDyAiHgAEfEAIuIBRMQDiIgHEBEPICIeQEQ8gIh4AJGm\n36q9e/du8eb48ePRrUajUbz5+/dvdGvbtm3RjtYxMjJSvLl+/Xp068ePH8Wbr1+/Rrd27twZ7ebz\n5AFExAOIiAcQEQ8gIh5ARDyAiHgAEfEAIuIBRMQDiIgHEBEPIFJNXjYD8OQBRMQDiIgHEBEPICIe\nQEQ8gIh4ABHxACLiAUTEA4iIBxARDyDS9DfGDQ0NFb9Z9+3bt+jW58+fizd79+6Nbj158qR4c/78\n+Wp0jEXp2bNnxZ/t7u7u6Fa1Wv7Rqdfr0a3R0dHizcDAwIJf0JMHEBEPICIeQEQ8gIh4ABHxACLi\nAUTEA4iIBxARDyAiHkCk6XdbhoeHizeTk5PRrVqtVrzZunVrdKu9vT3a0TqSz+nTp0+jW2vWrCne\nHD16NLrV1dUV7ebz5AFExAOIiAcQEQ8gIh5ARDyAiHgAEfEAIuIBRMQDiIgHEBEPICIeQKTpt2rv\n3LlTvDl8+HB0a2ZmpniTvK1YqVQqGzdujHa0jpcvXxZvXrx4Ed169OhR8Wb37t3RrTNnzhRvjhw5\nsuBnnjyAiHgAEfEAIuIBRMQDiIgHEBEPICIeQEQ8gIh4ABHxACLiAUSafjHu5MmTxZuxsbHo1r59\n+4o3s7Oz0a2pqanizebNm6NbLE49PT3Fm/3790e3jh07Vrz59OlTdGtiYiLazefJA4iIBxARDyAi\nHkBEPICIeAAR8QAi4gFExAOIiAcQEQ8gIh5ARDyASNNv1f78+bN4c+3atejW8uXLizfv3r2Lbk1P\nTxdvvFXbWvbs2VO8qdfr0a3kqyNfvXoV3ZqcnIx283nyACLiAUTEA4iIBxARDyAiHkBEPICIeAAR\n8QAi4gFExAOIiAcQqTYajX/6dwD+hTx5ABHxACLiAUTEA4iIBxARDyAiHkBEPICIeAAR8QAi4gFE\nxAOIiAcQEQ8gIh5ARDyAiHgAEfEAIuIBRMQDiPwXbXQIf/jWK8YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc0ce225a90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "weights = net['conv1'].W.get_value()\n",
    "\n",
    "plt.figure()\n",
    "f, axarr = plt.subplots(3,2)\n",
    "for i in range(3):\n",
    "    for j in range(2):\n",
    "        index = i*2+j\n",
    "        axarr[i,j].imshow(weights[index].reshape(5,5), cmap='Greys_r', interpolation='nearest')\n",
    "        axarr[i,j].axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other things you can easily do\n",
    "\n",
    "* Add things to the cost function. Just add something to the \"loss\" variable that is used for defining the updates. Theano will take care of calculating the derivates with respect to the inputs\n",
    "* Obtaining the representation from intermediate layers:\n",
    "\n",
    ">output_at_layer_fc1 = lasagne.layers.get_output(fc1)\n",
    "\n",
    ">get_representation = theano.function([input_var], output_at_layer_fc1)\n",
    "\n",
    "\n",
    "* Augmenting the input - easy to do since you have direct control of what is being fed to the training function\n",
    "* Using any type of learning rate schedule - you can define the learning rate as a theano shared variable, and set its value at any time in the computation (e.g. multiple by a value after each epoch, etc).\n",
    "* Use it for fine-tuning. By default the weights are initialized for you, but you can use pre-trained weights if you have them:\n",
    "> conv1 = lasagne.layers.Conv2DLayer(data, num_filters=32, filter_size=5, **W=pretrainedW, b=pretrainedB**)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
